Concerning the problem that traditional crack detection algorithm based on percolation model has low efficiency and detection results are prone to fracture, a crack detection algorithm based on multi-factor decision and percolation model was proposed. Firstly, an improved algorithm of accelerating crack inspection based on percolation model was proposed, which improves the efficiency of percolation processing by reducing a large number of redundant pixel points involved in percolation processing. Secondly, the extracted percolation points were used to percolation processing. Finally, a multi-factor decision connection algorithm based on crack orientation was proposed. In the algorithm, the rationality of crack connection was analyzed by four decision factors to improve the accuracy of crack connection. Different morphological crack images with different interfering objects in background were used in experiments. Compared with traditional percolation model detection algorithm and original algorithm of accelerating crack inspection based on percolation model and skeleton connection algorithm, the number of percolation points of the proposed algorithm was reduced by an average of 99.7% and 38.1%, respectively. The precision was increased by an average of 60.5% and 6.4%, respectively, and the recall was increased by an average of 10.5% and 4.0%, respectively. The experimental results show that the proposed algorithm can significantly improve the efficiency of percolation processing and improve the accuracy of crack detection.